Localized activity profiles and storage capacity of rate-based autoassociative networks 
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We study analytically the effect of metrically structured connectivity on the behavior of autoas- 
sociative networks. We focus on three simple rate-based model neurons: threshold-linear, binary 
or smoothly saturating units. For a connectivity which is short range enough the threshold-linear 
network shows localized retrieval states. The saturating and binary models also exhibit spatially 
modulated retrieval states if the highest activity level that they can achieve is above the maximum 
activity of the units in the stored patterns. In the zero quenched noise limit, we derive an ana- 
lytical formula for the critical value of the connectivity width below which one observes spatially 
non-uniform retrieval states. Localization reduces storage capacity, but only by a factor of 2 ~ 3. 
The approach that we present here is generic in the sense that there are no specific assumptions on 
the single unit input-output function nor on the exact connectivity structure. 



Recurrent neuronal networks, when endowed with as- 
sociative synaptic plasticity, are able to learn patterns 
of activity and retrieve them later when provided with 
partial cues - a property called autoassociative retrieval. 
This is believed to be an important ability of neocor- 
tical, as well as of hippocampal networks y.]. In the 
hippocampus, where several experiments are currently 
aimed at demonstrating attractor dynamics 2], it is the 
CA3 subfield which is thought to operate as a recur- 
rent autoassociative memory, and its recurrent connec- 
tions could be modeled, to a crude approximation, as 
extending uniformly across the network. In the neocor- 
tex, instead, the metrical organization of the connectivity 
cannot be neglected: neurons close to each other in the 
cortex are much more likely to be connected, while this 
probability becomes very low with distance |3(. Even in 
simplified theoretical models, there are technical prob- 
lems that complicate the analysis of associative networks 
with non-uniform connectivity: the distance dependence 
in the connectivity forces one to introduce "field" order 
parameters in the model|4(; moreover, asymmetric con- 
nectivity makes unapplicable those methods of equilib- 
rium statistical mechanics which were originally used for 
solving the classical models of associative retrieval |5|. 

Recently there have been several studies on how struc- 
ture in the connectivity affects performance of recurrent 
networks 0,0, 01 IS El- Most of these studies approach the 
problem by means of simulations and by focusing on one 
particular model neuron and connectivity structure. In 
one of these studies Q we described an analytical treat- 
ment of the problem in the case of an associative network 
with threshold-linear (XX) units. Due to difficulties in 
solving the steady-state equations, we developed an ap- 
proximate method for calculating the storage capacity of 
such a network, which was in reasonable agreement with 
simulations. In this paper, instead, we derive the equa- 
tions that govern the steady-state properties of a generic 
rate-based associative network and introduce a numeri- 
cal method that can be used for solving these equations 



with arbitrary accuracy. We calculate accurately the re- 
sults that we earlier obtained approximately for the XX 
network and extend the same analysis to a network of 
binary units as well as to a model which includes firing 
rate saturation. As a result of short range connectivity, 
in the XX network retrieval states may appear as local- 
ized bumps of activity, while these localized solutions are 
absent in a network with — 1 binary units. The possi- 
bility of spatial modulation for retrieval states depends 
on the maximum rate of the units relative to their rates 
in the memory patterns, and on how this saturation rate 
is approached. 

Consider a network of N = 2X — > oo units, in which 
the firing rate of the unit located at position r is repre- 
sented by a variable v(r) > 0. We assume that each unit 
receives C — » oo inputs from the other units in the net- 
work. The 'Hebbian' learning rule we consider prescribes 
that the synaptic weight between units r and r' be given 
as: 

J(r, r') = -L J2 P(r; r') (r/» - a) «(r') - a) , (1) 

where (r) represents the activity of unit located at r in 
memory pattern /i and p(r; r') is a function depending 
just on \n — r[\s/i which gives the probability that two 
neurons one at r and the other at r' are connected to 
each other (r^ is a component of r). 

Each ?7 M (r) is taken to be a 'quenched variable' drawn 
independently from a distribution £>(??), with the con- 
straints r] > 0, (77) = (rj 2 ) = a, where () stands for the av- 
erage over the distribution p(rj) Here we concentrate 
on the binary coding scheme p(rf) = aS(r]— 1) + (1— a)5(r]), 
but the calculation can be easily extended to any prob- 
ability distribution. We further assume that the input 
(local field) to the unit located at r is given by: 
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where the first term enables the memories encoded in 
the weights to determine the dynamics. In the second 
term, Xk = Xfc[i?(r)] are variables which depend on the 
global properties of the network activity. An example is 
the mean activity of the network x = J drv(r) which, 
through the second term, might be used to regulate itself, 
so that at any moment in time it approaches sparsity a. 
Following Q , we start our analysis by defining as an order 
parameter the local overlap: 
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The pattern v is said to be retrieved if J drm u (r) = 
O(N). We denote as m(r) = m u (r) the local overlap 
with the pattern v to be retrieved by a partial cue. 

The activity of each unit is determined by its input- 
output transfer function v(r) = F(h(r)). In general, us- 
ing a self-consistent signal-to-noise analysis 0, [H| , the 
fixed-point equations for such a network reduce to: 

^(r 2 ;ri) = J o]yK(y 2 ] r)p(r'; n) 

+ J drdr / K(r 2 ;r)K(r;r / )p(r;ri) + ... 
K(v 2] v) = ^p{v 2 -v){j ZW[r]) = p(r 2 ;r)*(r) 
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p\r 2 ) = ^ j driA(r 2 ;ri)/ 3 (ri) 
m(r 2 ) = ^ J drip(r 2 ;ri)/ 2 (ri) 
x k = X k [J DzG[y-T\] 
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where a = p/C is the storage load and: 

I 2 (r) = <(i7(r)/o-l) | £>*G[r ; r]> 

J 3 (r) = {j DzG[v-Tf) 

A(r 2 ;ri) = p(r 2 ; n) + 2p(r 2 ; ri)^(r 2 ; ri) 

+^(r 2 ;r!) 2 (5) 

and = dz^^\ while v(r) = G[r;T] = G[ft(r);T] is 

the self-consistent solution of v(r) = F(h(r) + r(r)i;(r)), 
and, finally, h(r) = ft(r) — r(r)i>(r) is the part of the local 
field at r which does not directly depend on v(r). 

If spatially modulated overlaps can be roughly de- 
scribed by the very first Fourier modes, then writing the 
above equations in Fourier space would help. For 
we focus, for simplicity, on a network which lies on a ID 
ring; the analysis could be extended, though, to arbitrary 
dimension. 

We write the connectivity matrix, m and p 2 in their 
Fourier modes and find the following fixed-point equa- 
tions (from now on c(x) = cos(^), s(x) = sin(j) and 



A is a dummy function label that can be c, representing 
a cosine, or s, representing a sine): 
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where we have: 
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and: 
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Assume that the b term in Eq|2| depends only on the 
mean activity, and it keeps it constant and equal to a. 
The above equations for m r , p r , ip°, ... can now be solved 
iteratively. The number of terms that one includes in the 
sum in EqEI (defining ip^ s ), together with the number of 
modes that one considers to approximate the connectiv- 
ity structure, determine the accuracy of the calculation. 

Let us now concentrate on two specific input-output 
functions. For the TL and the binary (B) transfer func- 
tions we have: 



G[r,r] 



TL 



-(h(r) - Th)e(h(r) - Th) 
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G[r,T] B = e(h(r)). (9) 

To proceed further we also assume a Gaussian connectiv- 
ity: p(r 2 - n) = (C/V27RT 2 ) exp[-(r 2 - r x ) 2 /27rt 2 ]. In 
Fig 0] we plot the amplitude of the first Fourier mode m\ 
- an indication of the deviation from the uniform solu- 
tion - as a function of a for TL units (black a = 0, blue 
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a = 0.1; in this figure and others always a = 0.2 and 
C/N = 0.05) and for B units (cyan and green curves, 
see below). With TL units, for small enough a/L the 
solution is essentially a localized bump. One can also see 
that decreasing quenched noise changes the transition to 
non-uniform retrieval from a smooth to an abrupt one. 
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FIG. 1: rhi versus a for g = 0.5, TL units and a = (black) 
and a = 0.1 (blue). The cyan curve is for — 1.5 binary units 
and green for — 2 B units, both at a = 0. 

— 1 binary networks fail to exhibit non-uniform re- 
trieval (Fig[I] p and |£j). The reason is simply related to 
setting the sparsity, and it can be understood intuitively 
as follows. There are two conditions to be satisfied for the 
retrieval state to exist: mo = J dr(r]^{r)/a — l)v(r) = 
0(1) and x = J drv(r) = a. The second means that, 
for spatially modulated retrieval states, in some parts of 
the network units with activity 1 in the corresponding 
stored pattern should have activity below 1 , and in other 
parts above 1. The latter requirement poses no problem 
to the TL network, whose units can reach high levels of 
activity. For a network with binary units, or with units 
that saturate, the crucial issue is whether the up state, or 
the saturation level, is sufficiently above 1 (the arbitrar- 
ily set activity level of active units in the stored patterns; 
obviously the argument can be generalized to non-binary 
stored patterns). Thus binary units with activity levels, 
say, and 1.5 (relative to the up state in the stored pat- 
tern) should be able to show spatially modulated activity 
profiles, although, rather than localized bumps, they ap- 
pear as square-shaped spatially restricted activity. This 
results in the cyan and green curves for rh\ in FigOJ 

To further assess the effect of the saturation level on 
the formation of localized retrieval states we consider the 
following input-output function: 

F(x) = e txnh(gx/e)Q(x). (10) 

One should notice that g is the slope at threshold and 
that for a sufficiently high e this is effectively just a TL 
function. For simplicity we focus on the a — » limit, 
as we do not expect the quenched noise to make any 
qualitative change in the behavior of the system, except 
for the smoothness of the transition. Fig. 2 shows how 
fhi changes with a for fixed g = 0.5 and different values 
of saturation, as measured by e. When the saturation 
is set at e = 1, for the intuitive reason above the first 



Fourier mode does not differ from zero. By increasing 
£, however, one approaches the TL regime. In Fig. 3 we 
plot rhi versus a for different values of g and both TL 
and saturating input-output functions. Notice the quasi- 
linear behavior for values of a below the transition. 
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FIG. 2: rhi versus a for g = 0.5 for different values of the 
saturation level: e = 2 (blue), e = 3 (black) and e = 4 (red). 
Cyan: TL units (e — > oo). 

When a = 0, analyzing the formation of no n- uniform 
solutions becomes simple even for a d-dimensional net- 
work. In this case, the fixed-point equations read: 

= i f dr II c ^n) < - l)F(h(r,rf)) > 

x = 1 J dr<F(h(r,rf)) > 
h(vrf) = C^--l)m(Y)-Th. (11) 

One can see that m^^^...^ = (1 — a)^ 1 o^ 2 o • • • di d o and 
Th = (1 — a)(l/a — 1) — F _1 (l) solve the above equa- 
tions, provided F _1 (l) < (1 — a) /a 0. However, for 
the following connectivity probability distribution: 

p(r, r') = f[ exp C^;'? 2 ) ( 12 ) 

(27TCT 2 )2 ^ 27RH 

this solution is stable only for a > <r c , where: 

cT c = ^^ln(a(l/a-l)2F'(F-i(l))) (13) 

and L is the half length of each dimension. For a < a c 
the uniform solution becomes unstable in the direction 
of the first Fourier mode mi...i. In a network of TL 
units the equation for fh\...\ - provided mi...i < a/[g(l — 
a)] - reads rai...i = g(pi...i/C)a(l/a — l) 2 mi...i, which 
at a c is satisfied for any mi...i. This means that the 
system is marginally stable at this point in the direction 
of resulting in a jump to mi...i = — &)] 

for a < <j c , as shown in the graphs. It is worth noting 
that such trivial equation for rai...i comes directly from 
the linear nature of the TL function above threshold. 
Adding quenched noise to a network of TL units or using 
any other function, e.g. the 2 introduced above, would 
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change this trivial equation to a non-linear form, with the 
disappearance of the jump, as evident in figures Fig^El 
and 03 
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FIG. 3: rhi versus a for different values of the gain: g = 0.45 
blue, g — 0.5 black, g — 0.55 red and g — 0.6 cyan, (upper 
panel) TL and (lower panel) saturating units with e — 2. 
With saturating units decreasing g (thus linearizing the input- 
output function close to threshold) sharpens the transition. 
The filled squares represent <j c as predicted by Ea ll3l 



We further investigate the effect of localized retrieval 
on the storage capacity of the network. In Fig^I we 
plot mo as a function of a for a = 500 and 800. Even 
though the storage capacity (a c = lnf a {a\rho(a) = 0}) 
decreases, the decrease is not too severe, even for very 
localized solutions. Thus, the maximum number of re- 
trievable patterns remains proportional to C. 




FIG. 4: mo versus a for different values of a in a TL network 
with g = 0.5. Black for a = 500, blue for a = 800 and red 
for structure-less network. Note the corresponding values: 
mi 1.22 for a 500, mi 0.8 for a 800 and mi = for 
the structure- less network, all when a = 0.05. 

The results presented here show that in general, a net- 
work with realistic single unit input-output transfer func- 
tion becomes capable of localized retrieval simply by ma- 
nipulating single unit saturation and linear gain. Increas- 
ing the gain, and/or the saturation level, makes retrieval 
states more localized. These parameters can be effec- 
tively controlled via inhibitory mechanisms. The effect 



of the quenched noise is minor on the qualitative behav- 
ior of the system, making the analytic formula Eq^| a 
reasonable approximation for a wide range of parame- 
ters. The fact that for a given value of a, changing the 
saturation level or the slope at threshold (g) can put the 
network out of the spatially modulated retrieval regime 
may explain the result of 

Localized retrieval, while quantitatively decreasing lo- 
cal storage capacity, may considerably increase the com- 
putational power of a network with structured connec- 
tivity. This can be appreciated by noting that in a large 
network, more than one memory pattern of activity may 
be retrieved at the same time, each in a different loca- 
tion, without much interference. A combination of locally 
retrieved memories can be thought of as a global, com- 
posite memory pattern. The number of such composite 
patterns would be combinatorially large, thus hugely in- 
creasing the overall storage capacity of the network. 

Moreover, while sparsity measures e.g. in IT cortex 
tend to yield high values (such as a ~ 0.7 [HEU) - seem- 
ingly in contradiction with the notion that associative 
networks require sparse coding in order to operate with 
a viable storage capacity - our result suggests another 
perspective. The reported values of a may have been 
measured, effectively, conditional to the recorded units 
being active in a localized retrieval state, thus 'overesti- 
mated' by neglecting the large silent part of the network. 

Each neuron in the neocortex receives of the order of 
10 4 synapses, and this number regulates a similar number 
of locally retrievable patterns. The fact that the number 
of memories stored in the neocortex seems much higher 
may stem from the combinatorial character of global 
memory patterns, allowed by the localization discussed 
here. 
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